Sensitivity of DFIT Tests of Measurement Invariance for Likert Data

Likert scales are routinely used in educational and psychological research as measures of constructs of interest. If sound scale development procedures are followed, the resulting scale can reliably and validly measure a construct. However, if a given scale is used to make comparisons among different populations of respondents (e.g., cultures; Riordan & Vandenberg, 1994), over time in longitudinal measurements (Golembiewski, Billingsley, & Yeager, 1976), or across different mediums of data collection (Ployhart, Weekley, Holtz, & Kemp, 2003), measurement invariance must be established before meaningful comparisons in observed data can be made (Raju, Laffitte, & Byrne, 2002; Taris, Bok, & Meijer, 1998; Vandenberg, 2002). Traditionally, confirmatory factor analytic (CFA) methods have been used to assess measurement invariance in Likert-type data. Recently however, IRT methods of establishing measurement invariance for these data have gained greater acceptance. IRT methods of establishing measurement invariance typically have used the nomenclature of differential item functioning (DIF). Originally developed for identifying biased test items in dichotomous data, DIF assessments have been adapted to polytomous data. DIF is said to occur when the relationship between levels of examinees’ latent trait (θ) and the probability of responses for a particular item differ between two groups (Camilli & Shepard, 1994). DIF can also be thought of as multidimensionality or as a factor affecting item responses other than participants’ θ level (Camilli & Shepard, 1994). For polytomous items, DIF can be thought of as differences in item true score functions across two groups (Cohen, Kim, & Baker, 1993). The Differential Functioning of Items and Tests (DFIT) framework (Raju, van der Linden, & Fleer, 1995) has been advanced for assessing both DIF and differential test functioning (DTF). Although the DFIT methodology is relatively new, it has been used in several studies published in prestigious journals (e.g., Collins, Raju, & Edwards, 2000; Donovan, Drasgow, & Probst, 2000; Ellis & Mead, 2000; Facteau & Craig, 2001; Flowers, Oshima, & Raju, 1999; Maurer, Raju, & Collins, 1999; Raju, et al., 2002). Articles published in these journals (including Applied Psychological Measurement, Educational and Psychological Measurement, and Journal of Applied Psychology), can have considerable influence on researchers undertaking similar studies and lend a high level of credibility to the DFIT methodology. Despite its high profile use in the past few years, the DFIT methodology is still under development. Few Monte-Carlo studies have been published that examine the efficacy of the DFIT program for detecting DIF in Likert data. In this study, we simulated data with known DIF in order to determine the efficacy of the DFIT program for detecting DIF and DTF.